{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "W_tvPdyfA-BL"
      },
      "source": [
        "##### Copyright 2018 The TensorFlow Authors."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "cellView": "form",
        "colab": {},
        "colab_type": "code",
        "id": "0O_LFhwSBCjm"
      },
      "outputs": [],
      "source": [
        "#@title Licensed under the Apache License, Version 2.0 (the \"License\");\n",
        "# you may not use this file except in compliance with the License.\n",
        "# You may obtain a copy of the License at\n",
        "#\n",
        "# https://www.apache.org/licenses/LICENSE-2.0\n",
        "#\n",
        "# Unless required by applicable law or agreed to in writing, software\n",
        "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
        "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
        "# See the License for the specific language governing permissions and\n",
        "# limitations under the License."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "PWUmcKKjtwXL"
      },
      "source": [
        "# Hub with Keras\n",
        "\n",
        "\u003ctable class=\"tfo-notebook-buttons\" align=\"left\"\u003e\n",
        "  \u003ctd\u003e\n",
        "    \u003ca target=\"_blank\" href=\"https://www.tensorflow.org/tutorials/images/hub_with_keras\"\u003e\u003cimg src=\"https://www.tensorflow.org/images/tf_logo_32px.png\" /\u003eView on TensorFlow.org\u003c/a\u003e\n",
        "  \u003c/td\u003e\n",
        "  \u003ctd\u003e\n",
        "    \u003ca target=\"_blank\" href=\"https://colab.research.google.com/github/tensorflow/docs/blob/master/site/en/tutorials/images/hub_with_keras.ipynb\"\u003e\u003cimg src=\"https://www.tensorflow.org/images/colab_logo_32px.png\" /\u003eRun in Google Colab\u003c/a\u003e\n",
        "  \u003c/td\u003e\n",
        "  \u003ctd\u003e\n",
        "    \u003ca target=\"_blank\" href=\"https://github.com/tensorflow/docs/blob/master/site/en/tutorials/images/hub_with_keras.ipynb\"\u003e\u003cimg src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" /\u003eView source on GitHub\u003c/a\u003e\n",
        "  \u003c/td\u003e\n",
        "\u003c/table\u003e"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "crU-iluJIEzw"
      },
      "source": [
        "[TensorFlow Hub](http://tensorflow.org/hub) is a way to share pretrained model components. See [their site](https://tfhub.dev/) for a searchable listing of pre-trained models.\n",
        "\n",
        "This tutorial provides a quick walkthrough of how to use `tfhub` with `tf.keras`."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "CKFUvuEho9Th"
      },
      "source": [
        "## Setup"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "7RVsYZLEpEWs"
      },
      "source": [
        "### Imports"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "IuuiFCJKL33d"
      },
      "outputs": [],
      "source": [
        "!pip install tensorflow_hub"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "OGNpmn43C0O6"
      },
      "outputs": [],
      "source": [
        "from __future__ import absolute_import\n",
        "from __future__ import division\n",
        "from __future__ import print_function\n",
        "\n",
        "import tensorflow as tf\n",
        "import tensorflow_hub as hub\n",
        "\n",
        "from tensorflow.keras import layers\n",
        "\n",
        "tf.VERSION"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "ApBqGprKnyPm"
      },
      "source": [
        "### Dataset\n",
        "\n",
        " For this example we'll use the TensorFlow flowers dataset: "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "yU3xd71Zx-dz"
      },
      "outputs": [],
      "source": [
        "import pathlib\n",
        "data_root = tf.keras.utils.get_file(\n",
        "  'flower_photos','https://storage.googleapis.com/download.tensorflow.org/example_images/flower_photos.tgz',\n",
        "   untar=True)\n",
        "\n",
        "data_root = pathlib.Path(data_root)\n",
        "print(data_root)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "W0U7QJDG0qRZ"
      },
      "outputs": [],
      "source": [
        "IMAGE_SIZE=224"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "DfjX-Qdan4hL"
      },
      "source": [
        "The simplest way to load this data into our model is using `tf.keras.preprocessing.image.ImageDataGenerator`:\n",
        "\n",
        "All of TensorFlow Hub's image modules expect input in the `[0, 1]`. Use the `ImageDataGenerator`'s `rescale` parameter to achieve this. "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "Dy8PqP5zunEx"
      },
      "outputs": [],
      "source": [
        "image_generator = tf.keras.preprocessing.image.ImageDataGenerator(rescale=1/255)\n",
        "image_data = image_generator.flow_from_directory(data_root, target_size=(IMAGE_SIZE, IMAGE_SIZE))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "tbzxwOEqoEDf"
      },
      "source": [
        "The resulting object is an iterator that returns `image_batch, label_batch` pairs."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "7pQ57Og5vNrf"
      },
      "outputs": [],
      "source": [
        "for image_batch,label_batch in image_data:\n",
        "  print(image_batch.shape)\n",
        "  print(label_batch.shape)\n",
        "  break"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "s4YuF5HvpM1W"
      },
      "source": [
        "## An ImageNet classifier"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "xEY_Ow5loN6q"
      },
      "source": [
        "### Download the classifier\n",
        "\n",
        "Use `hub.module` to load a mobilenet, and `tf.keras.layers.Lambda` to wrap it up as a keras layer."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "y_6bGjoPtzau"
      },
      "outputs": [],
      "source": [
        "mobilenet_layer = layers.Lambda(hub.Module(\"https://tfhub.dev/google/imagenet/mobilenet_v2_100_224/classification/2\"))\n",
        "mobilenet_classifier = tf.keras.Sequential([mobilenet_layer])"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "hCVnyAacog0P"
      },
      "source": [
        "### Run it on a batch of images\n",
        "\n",
        "TensorFlow hub requires that you manually ititialize it's vairables. "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "fIQkSI_Vt2Ne"
      },
      "outputs": [],
      "source": [
        "import tensorflow.keras.backend as K\n",
        "sess = K.get_session()\n",
        "init = tf.global_variables_initializer()\n",
        "\n",
        "sess.run(init)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "iNMdrnJzoppN"
      },
      "source": [
        "Now run the classifier on the image batch."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "7yiyuXJKt628"
      },
      "outputs": [],
      "source": [
        "result_batch = mobilenet_classifier.predict(image_batch)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "YrxLMajMoxkf"
      },
      "source": [
        "### Decode the predictions\n",
        "\n",
        "Fetch the `ImageNet` labels, and decode the predictions"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "ij6SrDxcxzry"
      },
      "outputs": [],
      "source": [
        "import numpy as np\n",
        "labels_path = tf.keras.utils.get_file('ImageNetLabels.txt','https://storage.googleapis.com/download.tensorflow.org/data/ImageNetLabels.txt')\n",
        "imagenet_labels = np.array(open(labels_path).read().splitlines())\n",
        "\n",
        "labels_batch = imagenet_labels[np.argmax(result_batch, axis=-1)]\n",
        "labels_batch"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "WqORYNSPv0-W"
      },
      "source": [
        "Now check how these predictions line up with the images:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "zqZPX67Zv0E0"
      },
      "outputs": [],
      "source": [
        "import matplotlib.pylab as plt\n",
        "\n",
        "plt.figure(figsize=(10,9))\n",
        "for n in range(30):\n",
        "  plt.subplot(6,5,n+1)\n",
        "  plt.imshow(image_batch[n])\n",
        "  plt.title(labels_batch[n])\n",
        "  plt.axis('off')\n",
        "_ = plt.suptitle(\"ImageNet predictions\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "n_oFmEk_rT4M"
      },
      "source": [
        "See the `LICENSE.txt` file for attributions.\n",
        "\n",
        "The results are far from perfect, but good considering that these are not the classes the model was trained for (except \"daisy\")."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "amfzqn1Oo7Om"
      },
      "source": [
        "## Simple transfer learning"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "JzV457OXreQP"
      },
      "source": [
        "### Download the headless model\n",
        "\n",
        "TensorFlow Hub also distributes models without the top classification layer. These can be used to easily do transfer learning."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "5wB030nezBwI"
      },
      "outputs": [],
      "source": [
        "mobilenet_features_module = hub.Module(\"https://tfhub.dev/google/imagenet/mobilenet_v2_100_224/feature_vector/2\")\n",
        "mobilenet_features = layers.Lambda(mobilenet_features_module)\n",
        "mobilenet_features.trainable = False"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "xNLcn5CesIMb"
      },
      "source": [
        "Again, this this must be initialized."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "HzBXXWjsyUGH"
      },
      "outputs": [],
      "source": [
        "init = tf.global_variables_initializer()\n",
        "sess.run(init)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "RPVeouTksO9q"
      },
      "source": [
        "### Attach a classification head\n",
        "\n",
        "Now wrap the hub layer in a `tf.keras.Sequential` model, and add a new classification layer."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "mGcY27fY1q3Q"
      },
      "outputs": [],
      "source": [
        "model = tf.keras.Sequential([\n",
        "  mobilenet_features,\n",
        "  layers.Dense(image_data.num_classes, activation='softmax')\n",
        "])"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "uppwC81ksYBx"
      },
      "source": [
        "Test run a single batch, to see that the result comes back with the expected shape."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "BgYI5G3ZFPWS"
      },
      "outputs": [],
      "source": [
        "result = model.predict(image_batch)\n",
        "result.shape"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "OHbXQqIquFxQ"
      },
      "source": [
        "### Train the model\n",
        "\n",
        "Use compile to configure the training process:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "3n0Wb9ylKd8R"
      },
      "outputs": [],
      "source": [
        "model.compile(\n",
        "  optimizer=tf.train.AdamOptimizer(), \n",
        "  loss='categorical_crossentropy',\n",
        "  metrics=['accuracy'])"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "58-BLV7dupJA"
      },
      "source": [
        "Now use the `.fit` method to train the model.\n",
        "\n",
        "Normally you would set the number of steps per epoch, but to keep this example short we'll reain just long enough to see that the loss is reducing."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "EyMDJxt2HdHr"
      },
      "outputs": [],
      "source": [
        "#steps_per_epoch=image_data.samples//image_data.batch_size)\n",
        "model.fit_generator(image_data, epochs=5, steps_per_epoch=10)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "Kd0N272B9Q0b"
      },
      "source": [
        "Now after, even just a few training iterations, we can already see that the model is making progress on the task."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "kb__ZN8uFn-D"
      },
      "source": [
        "### Decode the predictions\n",
        "\n",
        "To redo the plot from before, first get the ordered list of class names:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "nrkw6XDqELa8"
      },
      "outputs": [],
      "source": [
        "label_names = sorted(image_data.class_indices.items(), key=lambda pair:pair[1])\n",
        "label_names = np.array([key.title() for key, value in label_names])\n",
        "label_names"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "4Olg6MsNGJTL"
      },
      "source": [
        "Run the image batch through the model and comvert the indices to class names."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "Oq5Y06-d9xI_"
      },
      "outputs": [],
      "source": [
        "result_batch = model.predict(image_batch)\n",
        "\n",
        "labels_batch = label_names[np.argmax(result_batch, axis=-1)]\n",
        "labels_batch"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "CkGbZxl9GZs-"
      },
      "source": [
        "Plot the result"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "wC_AYRJU9NQe"
      },
      "outputs": [],
      "source": [
        "plt.figure(figsize=(10,9))\n",
        "for n in range(30):\n",
        "  plt.subplot(6,5,n+1)\n",
        "  plt.imshow(image_batch[n])\n",
        "  plt.title(labels_batch[n])\n",
        "  plt.axis('off')\n",
        "_ = plt.suptitle(\"Model predictions\")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "S9AVtdCqMSZ7"
      },
      "outputs": [],
      "source": [
        ""
      ]
    }
  ],
  "metadata": {
    "accelerator": "GPU",
    "colab": {
      "collapsed_sections": [
        "W_tvPdyfA-BL"
      ],
      "name": "Hub with Keras.ipynb",
      "private_outputs": true,
      "provenance": [],
      "toc_visible": true,
      "version": "0.3.2"
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}
